Thanks!

Wait for Python 3.11.8/3.12.2.

We do need to be able to work with all bug-fix versions that we can find out in the wild. .0 releases with bugs may be ignored, however we should have code that works with Python 3.11.7. So this shouldn't have to wait for a next CPython bug fix release - or did you mean waiting here for testing purposes?

Yeah, I think it would be nice to have some kind of test for this. I need to think more about how to set that up.

Why hasn't this been a problem so far? What changed?

Why do we need to wait for the new Python releases? The argument was added in Python 3.10.

Why hasn't this been a problem so far? What changed?

I identified BackgroundDownload.exe as the process preventing TemporaryDirectory from deleting certain files. It does not always happen, but calling Microsoft's Handle tool during the build process would occasionally give output like the following (see attempts 1 and 2 in this PR check):

------------------------------------------------------------------------------
BackgroundDownload.exe pid: 1304 fv-az1387-368\runneradmin
   40: File  (RW-)   D:\a\scipy\scipy\.mesonpy-n0tbbfzi\meson-private\cmake_scipy-openblas\CMakeFiles\CMakeScratch\TryCompile-s9yijw
  188: Section       \BaseNamedObjects\Cor_Private_IPCBlock_v4_1304
  18C: Section       \...\Cor_SxSPublic_IPCBlock
  284: File  (R--)   C:\Windows\assembly\pubpol11.dat
  288: File  (R-D)   C:\Program Files (x86)\Microsoft Visual Studio\Installer\resources\app\ServiceHub\Services\Microsoft.VisualStudio.Setup.Service\Microsoft.VisualStudio.Setup.dll

A new Windows Server 2019 runner image (the version used for these PR checks) was released on January 8th, which may coincide with the first reports of this issue. Interestingly, the BackgroundDownload.exe process disappears along with the failed cleanup errors when building with Windows Server 2022.

The weird behavior of BackgroundDownload.exe exposed an infinite recursion in the TemporaryDirectory cleanup code. That bug should be fixed in the next releases of Python 3.11/3.12.

I've modified the PR to make it more targeted while maintaining all current behavior/compatibility. After some more thought, I've also changed my mind about waiting to test this, because the relevant bugs are all beyond the scope of this repo. We can merge whenever ready.

IIUC, BackgroundDownload.exe is used to download Visual Studio updates. I don't understand why it should be preventing the cleanup of the temporary build directory. Please file a bug with GitHub to fix their images and with Microsoft for fixing their tools. I don't think we should work around this, especially if it means leaving temporary directories around.

If you need to work around this, you can simply pass a build directory explicitly with the build-dir build option. This disables the build directory cleanup.

Also, if this is a widespread issue, pip would run into the same cleanup problem when building a package for an sdist for local isntallation: pip unpacks the sdist into a temporary directory, and meson-python creates a build directory into that directory. If meson-python cleanup of the temporary build directory fails, the cleanup of the pip temporarty directory also fails.

Good point about the build-dir workaround and the downstream effects of a build dir cleanup failure (our failures were with build but I'm sure pip would fail too)! I'll close this and open an issue on the GitHub actions image.